This is the validation notebook for the background subtraction step as part of calwebb_image2. This step takes in a set of images and a set of background observations. If more than one background observation is given, they are combined into a sigma clipped mean before being subtracted from each of the science data images. For more information on the pipeline step visit the links below.
Step description: https://jwst-pipeline.readthedocs.io/en/latest/jwst/background_step/description.html
Pipeline code: https://github.com/spacetelescope/jwst/tree/master/jwst/background
The page describing the algorithm and any details can be found here:
https://outerspace.stsci.edu/display/JWSTCC/Vanilla+Imaging+Background+Subtraction
Here are some common terms that will be used throughout the notebook
JWST: James Webb Space Telescope
MIRI: Mid-Infrared Instrument
MIRISim: Simulator that creates pipeline ready MIRI simulated data
This test is performed by creating a set of simulated data with multiple point sources located at random locations across the MIRI field of view. There is a set of 4 simulated images at four dithered positions with 300+ stars and a galaxy in the MIRI field of view (including Lyot). This test also takes in a set of four simulated background images, at four dithered positions, with around 40 fainter stars across the field of view and Lyot mask. All images will be processed through calwebb_detector1, and put into an association file to be run through calwebb_image2. This will tell the background step which are the science observations and which are the background observations so that it will do a sigma clipped mean of the background exposures, then subtract the mean from each of the science observations.
The notebook shows the images (background, science, averaged background, and background subtracted) through the course of the notebook to demonstrate how well the algorithm works.
The set of data used in this particular test were created with the MIRI Data Simulator (MIRISim). There is a set of 4 simulated images at four dithered positions with 300+ stars and a galaxy in the MIRI field of view (including Lyot). This test also takes in a set of four simulated background images, at four dithered positions, with around 40 fainter stars across the field of view and Lyot mask. All of the images were created with the F770W filter.
The following cell sets up a temporary directory (using python's tempfile.TemporaryDirectory()), and changes the script's active directory into that directory (using python's os.chdir()). This is so that, when the notebook is run through, it will download files to (and create output files in) the temporary directory rather than in the notebook's directory. This makes cleanup significantly easier (since all output files are deleted when the notebook is shut down), and also means that different notebooks in the same directory won't interfere with each other when run by the automated webpage generation process.
If you want the notebook to generate output in the notebook's directory, simply don't run this cell.
If you have a file (or files) that are kept in the notebook's directory, and that the notebook needs to use while running, you can copy that file into the directory (the code to do so is present below, but commented out).
#****
#
# Set this variable to False to not use the temporary directory
#
#****
use_tempdir = True
# Create a temporary directory to hold notebook output, and change the working directory to that directory.
from tempfile import TemporaryDirectory
import os
import shutil
if use_tempdir:
data_dir = TemporaryDirectory()
# Save original directory
orig_dir = os.getcwd()
# Move to new directory
os.chdir(data_dir.name)
# For info, print out where the script is running
print("Running in {}".format(os.getcwd()))
Running in /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpj3vctz8b
By default, the notebook template environment sets up its CRDS cache (the "CRDS_PATH" environment variable) in /grp/crds/cache. However, if the notebook is running on a local machine without a fast and reliable connection to central storage, it makes more sense to put the CRDS cache locally. Currently, the cell below offers several options, and will check the supplied boolean variables one at a time until one matches.
use_local_crds_cache is False, then the CRDS cache will be kept in /grp/crds/cacheuse_local_crds_cache is True, the CRDS cache will be kept locallycrds_cache_tempdir is True, the CRDS cache will be kept in the temporary directorycrds_cache_notebook_dir is True, the CRDS cache will be kept in the same directory as the notebook.crds_cache_home is True, the CRDS cache will be kept in $HOME/crds/cachecrds_cache_custom_dir is True, the CRDS cache will be kept in whatever is stored in the
crds_cache_dir_name variable.If the above cell (creating a temporary directory) is not run, then setting crds_cache_tempdir to True will store the CRDS cache in the notebook's directory (the same as setting crds_cache_notebook_dir to True).
import os
if 'CRDS_CACHE_TYPE' in os.environ:
if os.environ['CRDS_CACHE_TYPE'] == 'local':
os.environ['CRDS_PATH'] = os.path.join(os.environ['HOME'], 'crds', 'cache')
elif os.path.isdir(os.environ['CRDS_CACHE_TYPE']):
os.environ['CRDS_PATH'] = os.environ['CRDS_CACHE_TYPE']
print('CRDS cache location: {}'.format(os.environ['CRDS_PATH']))